Derecho testing of aux_clm is as expected other than this test:

RXCROPMATURITYSKIPGEN_Ld1097.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput

which failed at RUN. It passes in the previous alpha 19 tag off of ctsm5.3.082 and it passes in ctsm5.3.085. I'm resubmitting to see if that solves it.

Erik, if I'm looking in the correct directory
/glade/derecho/scratch/erik/tests_Actsm54CMIP719ctsm5385acl/RXCROPMATURITYSKIPGEN_Ld1097.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput.GC.Actsm54CMIP719ctsm5385acl_int.gddgen/run
then the cesm.log has this error:
Abort with message Specified netCDF file does not exist

Is it not finding this file surfdata_10x15_hist_1850_78pfts_c251022.all_crops_everywhere.nc?

That file exists, so is it failing to find a datm file, such as /glade/campaign/cesm/cesmdata/inputdata/atm/datm7/CO2/fco2_datm_global_simyr_1750-2014_CMIP6_c180929.nc?

That one also exists. I think the name of the file that doesn't exist is not appearing in the log files...

It looks like the problem is due to #2930 which I just closed and then realized it needed to be reopened. The problem occurs at runtime rather than preview_namelist time right now.

My inclination is to merge this PR without fixing this? But, it seems like we should bring in a fix to #2930 and #3590 on b4b-dev. What do you think @slevis-lmwg?

Oh, I see, this file path has 258 characters: /glade/derecho/scratch/erik/tests_Actsm54CMIP719ctsm5385acl/RXCROPMATURITYSKIPGEN_Ld1097.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput.GC.Actsm54CMIP719ctsm5385acl_int.gddgen/surfdata_10x15_hist_1850_78pfts_c251022.all_crops_everywhere.nc
In other words if your /tests directory had a default name (e.g. tests_1121-130823de) then the test would PASS.

I recommend running the same test manually (using ./create_test RX... -c <old_baseline> -g <new_baseline) to confirm that it passes and to end up with the correct TestStatus in the new baseline.

Regarding your second question, I would start by adding milestones to these two issues.

I had trouble getting the failing test to run by hand. But, got it to work by running a single test from run_sys_tests. The by hand tests at failed at runtime with problems recognizing the ctsm_pylib environment. I think this tells me that the ctsm_pylib isn't inherited into runtime for at least these tests. I'm not sure why. But, something that we may need to look into.

I'm going to bypass the requirement for a review (since this is just an update), to expedite this. It's Thanksgiving eve, and I'm not sure I can get someone to do a quick review. And with a branch update like this there doesn't seem to be enough of a need for it.

Erik, I can review/approve.

I had trouble getting the failing test to run by hand. But, got it to work by running a single test from run_sys_tests. The by hand tests at failed at runtime with problems recognizing the ctsm_pylib environment. I think this tells me that the ctsm_pylib isn't inherited into runtime for at least these tests. I'm not sure why. But, something that we may need to look into.

Worth opening an issue. I wonder whether it's an issue specific to this test, because I have not had trouble submitting ./create_test successfully recently.